Image forming apparatus, image forming method and image sensor

ABSTRACT

An exemplary image forming apparatus includes: an illumination system which sequentially emits illuminating light beams from multiple different irradiation directions; an image sensor which captures a plurality of different images in the multiple different irradiation directions, respectively; an image processing section which forms a high-resolution image; and a memory which stores data about the ratio of light rays that have been incident on a photoelectric conversion section of each pixel of the image sensor to light rays that have passed through the upper surface of a plurality of subpixels included in the pixel with respect to each of the multiple irradiation directions. The image processing section forms the high-resolution image based on the data that has been retrieved from the memory by extracting, as a vector, a set of pixel values associated with the multiple irradiation directions from pixel values that form each of the plurality of images.

This is a continuation of International Application No.PCT/JP2014/002985, with an international filing date of Jun. 5, 2014,which claims priority of Japanese Patent Application No. 2013-121112,filed on Jun. 7, 2013, the contents of which are hereby incorporated byreference.

BACKGROUND

1. Technical Field

The present application relates to an image forming apparatus, imageforming method and image sensor.

2. Description of the Related Art

A two-dimensional image sensor in which a lot of photoelectricconversion sections are arranged in columns and rows within its imagingsurface has been used as an image sensor for an image capture device.Each of those photoelectric conversion sections is typically aphotodiode which has been formed on a semiconductor layer or on asemiconductor substrate, and generates electric charges based on thelight incident thereon. The resolution of the two-dimensional imagesensor depends on the arrangement pitch or density of the photoelectricconversion sections on the imaging surface. However, since thearrangement pitch of the photoelectric conversion sections has becomealmost as short as the wavelength of visible radiation, it is verydifficult to further increase the resolution.

An image captured by the image sensor is comprised of a lot of pixels,each of which is defined by a unit region including a singlephotoelectric conversion section. Since there is an area to be occupiedby wiring on the imaging surface, the photosensitive area R2 of a singlephotoelectric conversion section is smaller than the area R1 of a singlepixel. The ratio (R2/R1) of the photosensitive area R2 to the area R1 ofeach pixel is called an “aperture ratio”, which may be approximately25%, for example. If the aperture ratio is low, the amount of light thatcan be used for photoelectric conversion decreases, and therefore, thequality of a pixel signal to be output by the image sensor declines.However, by adopting a configuration in which an array of micro lensesis arranged to face the imaging surface and in which each of those microlenses faces, and converges light onto, its associated photoelectricconversion section, the photosensitive area R2 can be increased soeffectively that the aperture ratio (R2/R1) can be raised to thevicinity of one. Nevertheless, even if the aperture ratio (R2/R1) isincreased in this manner, the arrangement pitch and arrangement densityof pixels do not increase, and therefore, the resolution does notchange.

Japanese Laid-Open Patent Publication No. 2006-140885 discloses how toincrease the resolution by super-resolution technique. To increase theresolution by such a technique, restoration needs to be done bydeconvolution, and therefore, a point spread function (PSF) should beobtained. For example, to determine the PSF actually, a dotted lightsource needs to be used. That is why it has been proposed that the PSFbe obtained using quantum dots or fluorescence beads.

However, it is difficult to obtain the PSF accurately. In addition,since the magnitude of the PSF is proportional to the zoom power ofshooting, the measuring error of the PSF increases proportionally to thezoom power of shooting. As a result, the quality of the high-resolutionimage deteriorates proportionally to the resolution.

SUMMARY

One non-limiting, and exemplary embodiment provides a technique toachieve higher resolution.

In one general aspect, an image forming apparatus disclosed hereinincludes: an illumination system which sequentially emits illuminatinglight beams from multiple different irradiation directions with respectto an object and irradiates the object with the illuminating lightbeams; an image sensor which is arranged at a position where theilluminating light beams that have been transmitted through the objectare incident and which captures a plurality of different images in themultiple different irradiation directions, respectively; an imageprocessing section which forms a high-resolution image of the object,having a higher resolution than any of the plurality of images, based onthe plurality of images; and a memory which stores data about the ratioof light rays that have been incident on a photoelectric conversionsection of each pixel of the image sensor to light rays that have passedthrough the upper surface of a plurality of subpixels included in thepixel with respect to each of the multiple irradiation directions. Theimage processing section forms the high-resolution image of the objectbased on the data that has been retrieved from the memory by extracting,as a vector, a set of pixel values associated with the multipleirradiation directions from pixel values that form each of the pluralityof images.

According to embodiments of the present disclosure, the resolution canbe increased by synthesizing together a plurality of low-resolutionimages that have been captured by a single image sensor.

These general and specific aspects may be implemented using a system, amethod, a computer program, a computer-readable recording medium, and animage sensor, and any combination of systems, methods, computerprograms, computer-readable recording media, and image sensors.

Additional benefits and advantages of the disclosed embodiments will beapparent from the specification and Figures. The benefits and/oradvantages may be individually provided by the various embodiments andfeatures of the specification and drawings disclosure, and need not allbe provided in order to obtain one or more of the same.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a plan view schematically illustrating an exemplaryarrangement of photodiodes in an image sensor.

FIG. 2 is a plan view schematically illustrating the relation between asingle pixel and an aperture area in an image sensor.

FIG. 3 is a cross-sectional view schematically illustrating the relationbetween a single pixel and an aperture area in an image sensor.

FIG. 4A is a cross-sectional view schematically illustrating anexemplary configuration and operation of an image forming apparatusaccording to the present disclosure.

FIG. 4B is a cross-sectional view schematically illustrating anexemplary configuration and operation of an image forming apparatusaccording to the present disclosure.

FIG. 5A illustrates an exemplary illumination unit for an image formingapparatus according to the present disclosure.

FIG. 5B illustrates another exemplary illumination unit for an imageforming apparatus according to the present disclosure.

FIG. 5C illustrates still another exemplary illumination unit for animage forming apparatus according to the present disclosure.

FIG. 6 illustrates yet another exemplary illumination unit for an imageforming apparatus according to the present disclosure.

FIG. 7 is a cross-sectional view illustrating how a light ray may beincident on an image sensor according to the present disclosure.

FIG. 8 is a cross-sectional view illustrating how light rays may also beincident on an image sensor according to the present disclosure.

FIG. 9 is a cross-sectional view illustrating how light rays may also beincident on an image sensor according to the present disclosure.

FIG. 10 is a cross-sectional view illustrating how light rays may alsobe incident on an image sensor according to the present disclosure.

FIG. 11 is a cross-sectional view illustrating how light raystransmitted through an object may be incident on an image sensoraccording to the present disclosure.

FIG. 12A is a table summarizing relations between the output values A₁to A₄ of a photodiode 40 and transmittances S₁ to S₄ which were obtainedby capturing images in respective irradiation directions J1 to J4.

FIG. 12B is a table summarizing relations between the output values of aphotodiode 40 and transmittances S₁ to S₄ which were obtained bycapturing images in respective irradiation directions J1 to J4 withrespect to N pixels.

FIG. 13A is a table summarizing relations between the output values A₁to A₅ of a photodiode 40 and transmittances S₁ to S₄ which were obtainedby capturing images in respective irradiation directions J1 to J5.

FIG. 13B is a table summarizing relations between the output values of aphotodiode 40 and transmittances S₁ to S₄ which were obtained bycapturing images in respective irradiation directions J1 to J5 withrespect to N pixels.

FIG. 14 is a table summarizing relations between the output values A₁ toA₃ of a photodiode 40 and transmittances S₁ and S₂ which were obtainedby capturing images in respective irradiation directions J1 to J3.

FIG. 15 shows conditional expressions defining the relations shown inFIG. 14.

FIG. 16 is a graph showing three lines which are represented by thethree conditional expressions shown in FIG. 15 on a two-dimensionalcoordinate plane, of which the abscissa is S₁ and the ordinate is S₂.

FIG. 17 is a block diagram showing an example of general configurationfor an image forming apparatus according to a first embodiment.

FIG. 18 is a cross-sectional view illustrating how an error is caused inthe point of incidence of a light ray due to a positional shift of alight source in an image forming apparatus according to the firstembodiment.

FIG. 19 is a cross-sectional view illustrating how the point ofincidence of a light ray shifts due to the spread of a light beam whichhas been emitted from a point light source in an image forming apparatusaccording to the first embodiment.

FIG. 20 illustrates an exemplary range in which calculations need to bemade using a matrix.

FIG. 21 is a perspective view illustrating an exemplary image sensorwith an opaque portion.

FIG. 22 is a cross-sectional view illustrating an exemplary image sensorwith an opaque portion.

FIG. 23 is an exemplary flowchart showing how an image forming apparatusaccording to the first embodiment operates.

FIG. 24 is a block diagram showing an example of general configurationfor an image forming apparatus according to a second embodiment.

FIG. 25 is an exemplary flowchart showing how an image forming apparatusaccording to the second embodiment operates.

FIG. 26 illustrates how to form a zoomed-in image 2801 to obtain ahigh-resolution image at a zoom power of 2×.

FIG. 27 illustrates a modified example with a holder which holds anobject of shooting and an image sensor in an attachable and removablestate.

FIG. 28 illustrates another modified example with a holder which holdsan object of shooting and an image sensor in an attachable and removablestate.

FIG. 29 illustrates still another modified example with a holder whichholds an object of shooting and an image sensor in an attachable andremovable state.

FIG. 30 illustrates yet another modified example with a holder whichholds an object of shooting and an image sensor in an attachable andremovable state.

FIG. 31 illustrates yet another modified example with a holder whichholds an object of shooting and an image sensor in an attachable andremovable state.

FIG. 32 illustrates yet another modified example with a holder whichholds an object of shooting and an image sensor in an attachable andremovable state.

FIG. 33 illustrates yet another modified example with a holder whichholds an object of shooting and an image sensor in an attachable andremovable state.

FIG. 34 illustrates yet another modified example with a holder whichholds an object of shooting and an image sensor in an attachable andremovable state.

FIG. 35 illustrates yet another modified example with a holder whichholds an object of shooting and an image sensor in an attachable andremovable state.

FIG. 36 illustrates yet another modified example with a holder whichholds an object of shooting and an image sensor in an attachable andremovable state.

FIG. 37 illustrates yet another modified example with a holder whichholds an object of shooting and an image sensor in an attachable andremovable state.

FIG. 38 illustrates yet another modified example with a holder whichholds an object of shooting and an image sensor in an attachable andremovable state.

FIG. 39 illustrates yet another modified example with a holder whichholds an object of shooting and an image sensor in an attachable andremovable state.

DETAILED DESCRIPTION

Before embodiments of an image forming apparatus according to thepresent disclosure are described, an exemplary basic configuration foran image sensor will be described.

FIG. 1 is a plan view schematically illustrating a portion of theimaging surface of a CCD image sensor which is an exemplary image sensor113. As shown in FIG. 1, a number of photodiodes (photoelectricconversion sections) 40 are arranged in columns and rows on the imagingsurface. In FIG. 1, a single pixel 50 is indicated by the dottedrectangle. On the imaging surface, a lot of pixels 50 are denselyarranged in columns and rows.

The light that has been incident on each photodiode 40 generateselectric charges inside the photodiode 40. The amount of the electriccharges generated varies according to the amount of the light that hasbeen incident on that photodiode 40. The electric charges generated byeach photodiode 40 move to, and are sequentially transferred through, avertical charge transfer path 44 which runs vertically to enter ahorizontal charge transfer path 46. Next, the electric charges aretransferred through the horizontal charge transfer path 46 which runshorizontally and are output as a pixel signal to a device outside ofthis image sensor 113 through one end of the horizontal charge transferpath 46. Although not shown, transfer electrodes are arranged on thesecharge transfer paths 44 and 46. It should be noted that the imagesensor 113 for use in an image forming apparatus according to thepresent disclosure does not have to have this configuration. Forexample, the CCD image sensor may be replaced with an MOS image sensor.

In the imaging surface, the vertical arrangement pitch of thephotodiodes 40 does not have to agree with their horizontal arrangementpitch. In this description, however, the vertical and horizontalarrangement pitches of the photodiodes 40 are supposed to be equal toeach other and are both supposed to be K [μm] for the sake ofsimplicity.

FIG. 2 is a plan view schematically illustrating a single pixel 50 and aphotodiode 40 included in the pixel 50. In this example, the size ofeach pixel is K [μm]×K [μm], and the size of the photodiode 40 (i.e.,the size of its photosensitive area) is P [μm]×p [μm]. Thus, the area ofa single pixel is given by R1=K×K and the area of a single photodiode 40is given by R2=P×P (where “×” denotes multiplication). It should benoted that the resolution is determined in this embodiment by the sizeof the photodiode 40 (i.e., the size of its photosensitive area), not bythe pixel pitch. Considering the wavelength of visible radiation for useas illuminating light, however, the size P of the photodiode 40according to this embodiment may be set to be equal to or greater than0.1 μm.

In the image forming apparatus of the present disclosure, no microlenses are provided for each photodiode 40. That is why the rest of eachpixel 50 other than the photosensitive area (i.e., the area with thesize P×P) of the photodiode 40 is an opaque area. The light incident onthe opaque area is not converted into electric charge and does notgenerate any pixel signal, either. The photosensitive area indicated byP [μm]×P [μm] may be called an “aperture area”. The location, shape andsize of the photodiode 40 in each pixel 50 do not have to be theexemplary ones illustrated in FIG. 2.

The pixel region and photodiode typically have a rectangular shape onthe imaging surface. In that case, supposing n and m are real numbers,the ratio of the photodiode's size to the pixel region's size asmeasured horizontally in the imaging surface can be represented by(1/n), and the ratio of the photodiode's size to the pixel region's sizeas measured vertically in the imaging surface can be represented by(1/m). Then, the aperture ratio can be represented by (1/n)×(1/m), wheren and m may both be real numbers which are equal to or greater than two.

FIG. 3 is a cross-sectional view schematically illustrating an exemplarycross-sectional structure for a single pixel 50 included in the imagesensor 113. As shown in FIG. 3, the image sensor includes asemiconductor substrate 400, a photodiode (PD) 40 which has been formedon the surface of the semiconductor substrate 400, an interconnect layer402 supported on the semiconductor substrate 400, an opaque layer 42which covers the interconnect layer 402, and a transparent layer 406which covers the light incident side of the semiconductor substrate 400.Since FIG. 3 illustrates a cross section of a portion corresponding to asingle pixel, only one photodiode 40 is shown in FIG. 3. Actually,however, a huge number of photodiodes 40 are arranged on the singlesemiconductor substrate 400. If the image sensor 113 is a CCD imagesensor, a doped layer (not shown) functioning as a vertical orhorizontal charge transfer path is provided under the interconnect layer402 in the semiconductor substrate 400. The interconnect layer 402 isconnected to an electrode (not shown) which is arranged on the chargetransfer path. If the image sensor 113 is an MOS image sensor, MOStransistors (not shown) are arranged on a pixel-by-pixel basis on thesemiconductor substrate 400. Each of those MOS transistors functions asa switching element to extract electric charges from its associatedphotodiode 40.

Every component of the image sensor 113 but the photodiode 40 is coveredwith the opaque layer 42. In the example illustrated in FIG. 3, theregion covered with the opaque layer 42 is filled in black.

The image sensor for use in this embodiment does not have to have such aconfiguration but may also be a CCD or MOS image sensor of a backsideillumination type, for example.

Next, an exemplary general configuration for an image forming apparatusaccording to the present disclosure will be described with reference toFIGS. 4A and 4B.

The image forming apparatus illustrated in FIGS. 4A and 4B includes anillumination unit 111 which sequentially emits illuminating light beamsfrom multiple different light source directions (irradiation directions)with respect to an object 30 and irradiates the object 30 with theilluminating light beams, and an image sensor 113 which is arranged at aposition where the illuminating light beams that have been transmittedthrough the object 30 are incident and which captures a plurality ofdifferent images in the multiple different irradiation directions,respectively. This image forming apparatus further includes an imageprocessing section 12 which forms a high-resolution image based on theplurality of images that have been captured in the multiple differentirradiation directions. This image processing section 12 can form ahigh-resolution image of the object which has a higher resolution thanany of the plurality of images provided by the image sensor 113. Theimage processing section 12 may be implemented as either ageneral-purpose computer or a dedicated computer.

When the image sensor 113 is going to capture a first image (see FIG.4A), the illuminating unit 111 makes an illuminating light beam incidenton the object 30 from a first direction. On the other hand, when theimage sensor 113 is going to capture a second image (see FIG. 4B), theilluminating unit 111 makes an illuminating light beam incident on theobject 30 from a second direction. Among the light rays illustrated inFIGS. 4A and 4B, the ones incident on the opaque layer 42 are not usedto capture any image. In other words, only the light rays that have beenincident on the photodiode 40 are used to capture images among the lightrays that have been emitted from the illuminating unit 111.

If the direction in which an incoming light ray is incident on theobject 30 changes significantly, the light ray may have been transmittedthrough different regions of the object 30 before being incident on thephotodiode 40. In the image forming apparatus of this embodiment,however, the plurality of irradiation directions may be adjusted so thatwhile the image sensor 113 is capturing a plurality of images, at leastsome of the light rays that have been transmitted through the sameportion of the object 30 are incident on the photoelectric conversionsection of the image sensor 113. It should be noted that the object 30that can be shot by the image forming apparatus of the presentdisclosure is a matter, at least a part of which is a region that cantransmit a light ray. For example, the object 30 may be a slide plateincluding a pathological sample with a thickness of several μm. Theobject 30 does not have to have a plate shape but may also be powder orliquid as well. When measured along a normal to the imaging surface, theobject 30 may have a size of 2 μm or less, for example.

Next, a first exemplary configuration for the illumination unit 111 willbe described with reference to FIGS. 5A, 5B and 5C.

The illumination unit 111 with this first exemplary configurationincludes a plurality of light sources (illuminating light sources) 10 a,10 b and 10 c, which are arranged at respectively different positionscorresponding to multiple different irradiation directions and areturned ON sequentially. For example, when the light source 10 a isturned ON, light is emitted from the light source 10 a and irradiatesthe object 30 as shown in FIG. 5A. In FIGS. 5A to 5C, the light emittedfrom the light sources 10 a, 10 b and 10 c is illustrated as if thelight was diverging. Actually, however, the distances from the lightsources 10 a, 10 b and 10 c to the image sensor 113 are so long that thelight incident on the object 30 and image sensor 113 may be regarded assubstantially parallel. Optionally, the light radiated from the lightsources 10 a, 10 b and 10 c may also be converged by an optical systemsuch as a lens (not shown) into a parallel light beam or aquasi-parallel light beam. That is why the light sources 10 a, 10 b and10 c may be either point light sources or surface-emitting lightsources. The object 30 is put on the upper surface of the image sensor113. The upper surface of the image sensor 113 is indicated by thedashed line in FIG. 5A and functions as an object supporting portion112.

First, an image is captured by the image sensor 113 while the object 30is irradiated with the light emitted from the light source 10 a. Next,the light source 10 b, for example, is turned ON and the light sources10 a and 10 c are turned OFF. In this case, light is emitted from thelight source 10 b and irradiates the object 30 as shown in FIG. 5B. Insuch a state, an image is captured by the image sensor 113 while theobject 30 is irradiated with the light emitted from the light source 10b. Next, the light source 10 c is turned ON and the light sources 10 aand 10 b are turned OFF. In this case, light is emitted from the lightsource 10 c and irradiates the object 30 as shown in FIG. 5C. In such astate, another image is captured by the image sensor 113.

In the examples illustrated in FIGS. 5A to 5C, the object 30 isirradiated with light beams coming from three different irradiationdirections, and an image is captured every time the object 30 isirradiated with a light beam. Thus, three images are captured in total.However, the number of light sources that the illumination unit 111 hasdoes not have to be three. Optionally, multiple light sources withrespectively different emission wavelengths may be arranged close toeach other in the same irradiation direction. For example, if lightsources which emit red, green and blue light beams (which will behereinafter referred to as “RGB light sources”) are arranged at and nearthe position of the light source 10 a shown in FIG. 5A, three images canbe captured by sequentially radiating the red, green and blue lightbeams in the state shown in FIG. 5A. And once those three images arecaptured, a full-color image can be obtained just by superposing thoseimages one upon the other. These images are time-sequential colorimages. In this case, if the RGB light sources are turned ONsimultaneously, these light sources can function as a single white lightsource. Also, if color filters are provided for the image sensor, awhite light source may be used as the light source.

It should be noted that the wavelength of the light sources that theillumination unit 111 has does not have to fall within the visibleradiation range but may also fall within the infrared or ultravioletrange as well. Alternatively, white light may be emitted from each ofthose light sources. Still alternatively, cyan, magenta and yellow lightbeams may be emitted from those light sources.

Next, look at FIG. 6, which schematically illustrates a second exemplaryconfiguration for the illumination unit 111. In the exemplaryconfiguration shown in FIG. 6, the illumination unit 111 includes atleast one light source 10 which is supported so as to be movable to anydirection. By moving this light source 10, light can be emitted from anyof multiple irradiation directions and can irradiate the object 30.

It should be noted that even in the examples illustrated in FIGS. 5A to5C, the light sources 10 a, 10 b and 10 c do not have to be fixed atparticular positions but may also be supported movably. Alternatively, alight beam emitted from a single fixed light source 10 may have itsoptical path changed by an actuated optical system such as a movablemirror so as to be incident on the object 30 from a different direction.

In the examples illustrated in FIGS. 5A to 5C and in the exampleillustrated in FIG. 6, the irradiation direction is supposed to changewithin a plane which is parallel to the paper. However, the irradiationdirection may also define a tilt angle with respect to that plane. Inthe following description, however, zooming (i.e., resolutionenhancement) along one-dimensional direction on the imaging surface willbe described for the sake of simplicity. In 2-dimensional resolutionenhancement, If the intended zoom power is n (where n is an integer thatis equal to or greater than two), n² light sources may be used, forexample.

It should be noted that the “irradiation direction” of illuminatinglight is determined by the relative arrangement of its light source withrespect to the object (or imaging surface). In this description, theimaging surface is regarded as a reference plane and the direction fromwhich an illuminating light ray has come before being incident on theimaging surface is defined to be the “irradiation direction”. Supposingthe horizontal and vertical directions on the imaging surface are X andY axes, respectively, and a normal to the imaging surface is Z axis, theirradiation direction may be determined by a vector in the XYZcoordinate system. The irradiation direction may be an arbitrary one, sois the number of irradiation directions.

The irradiation direction that is perpendicular to the imaging surfacemay be represented by the vector (0, 0, 1). If the interval between theimaging surface and the object is L, sixteen different irradiationdirections θ1 through θ16 may be represented by the vectors (0, 0, L),(K/4, 0, L), (2K/4, 0, L), (3K/4, 0, L), (0, K/4, L), (K/4, K/4, L),(2K/4, K/4, L), (3K/4, K/4, L), (0, 2K/4, L), (K/4, 2K/4, L), (2K/4,2K/4, L), (3K/4, 2K/4, L), (0, 3K/4, L), (K/4, 3K/4, L), (2K/4, 3K/4, L)and (3K/4, 3K/4, L), respectively. Another angle at which the sameimages can be captured may also be adopted.

Next, the directions in which the illuminating light beams are incidentwill be described with reference to FIGS. 7 to 9, in each of which shownis a light beam incident on a photodiode 40 of interest.

First of all, look at FIG. 7. To illustrate the principle of resolutionenhancement, each pixel of the image sensor 113 is supposed to include aplurality of subpixels. The number of subpixel regions is not unique tothe structure of the image sensor 113 but may be set based on the numberof times the object is irradiated with light beams with its irradiationdirection changed while images are captured. In the example illustratedin FIG. 7, a single pixel 50 includes four subpixels 50 a, 50 b, 50 c,and 50 d. Since a single pixel includes a single photodiode 40,information about fine regions associated with those subpixels 50 a, 50b, 50 c, and 50 d cannot be obtained individually from the object 30 ina normal image capturing session. That is to say, in this description,the “plurality of subpixels included in each pixel of the image sensor”refers herein to multiple divisions of a single pixel region allocatedto each photodiode of the image sensor.

The object 30 may be arranged either in contact with, or close to, theupper surface 48 of the image sensor 113 while images are captured.Thus, the upper surface 48 of the image sensor 113 can function as anobject supporting portion (object supporting surface). Among therespective surfaces of the subpixels 50 a, 50 b, 50 c, and 50 d, theirsurface that forms part of the upper surface 48 of the image sensor 113is defined herein to be the “subpixel's upper surface”.

In the example illustrated in FIG. 7, four subpixels 50 a, 50 b, 50 c,and 50 d are arranged in a single pixel 50 along the X axis shown inFIG. 7. However, the subpixels may be arranged in any other arbitrarypattern. For example, the subpixels may be arranged two-dimensionallyaccording to the spread of the imaging surface. In this example, eachpixel is supposed to be comprised of four subpixels which are arrangedone-dimensionally in the X-axis direction for the sake of simplicity,and attention is paid to only one pixel.

Next, it will be described with reference to FIGS. 8 to 10 how thepercentage of an illuminating light ray that has been transmittedthrough the upper surface of a subpixel 50 a, 50 b, 50 c or 50 d andthen incident on the photodiode 40 changes with the direction from whichthat illuminating light ray has come (i.e., with its irradiationdirection). In the state illustrated in FIGS. 8 to 10, the object 30 hasnot been put on the image sensor 113 yet.

In FIG. 8, the illuminating light rays are incident perpendicularly ontothe imaging surface. Such illuminating light rays are obtained byirradiating the image sensor 113 with the light emitted from the lightsource 10 a shown in FIG. 5A, for example. In this case, the light raysthat have been transmitted through the respective upper surfaces of thesubpixels 50 a, 50 b, 50 c and 50 d are incident on the photodiode 40 atratios of ½, 1, ½ and 0, respectively. For example, supposing the amountof light transmitted through the upper surface of the subpixel 50 b andincident on the photodiode 40 is the unity (i.e., one), the amount oflight transmitted through the upper surface of the subpixel 50 a andincident on the photodiode 40 is ½. A half of the light transmittedthrough the upper surface of the subpixel 50 c is incident on thephotodiode 40, while the other half is incident on an opaque layer 42.And all of the light transmitted through the upper surface of thesubpixel 50 d is incident on the opaque layer 42, not on the photodiode40. It should be noted that the numerical values “½, 1, ½ and 0”defining the ratios described above are only an example. Alternatively,the ratios may be represented either by common fractions which use aninteger of 3 or more or by decimal fractions. If the image sensor 113has a different structure (e.g., if the image sensor 113 has a differentaperture ratio), these ratios may change.

Next, look at FIG. 9, in which the illuminating light rays are incidentobliquely onto the imaging surface. Such illuminating light rays areobtained by irradiating the image sensor 113 with the light emitted fromthe light source 10 c shown in FIG. 5C, for example. In this case, thelight rays that have been transmitted through the respective uppersurfaces of the subpixels 50 a, 50 b, 50 c and 50 d are incident on thephotodiode 40 at ratios of 1, ½, 0 and 0, respectively.

Next, look at FIG. 10, in which the illuminating light rays are incidentobliquely onto the imaging surface from the irradiation directionopposite from the one shown in FIG. 9. Such illuminating light rays areobtained by irradiating the image sensor 113 with the light emitted fromthe light source 10 b shown in FIG. 5B, for example. In this case, thelight rays that have been transmitted through the respective uppersurfaces of the subpixels 50 a, 50 b, 50 c and 50 d are incident on thephotodiode 40 at ratios of 0, 0, ½, and 1, respectively.

By adjusting the irradiation direction, light can be incident on theimage sensor 113 from an arbitrary one of multiple different directions.And the “ratio” described above is determined by choosing one of thosemultiple different irradiation directions. If the structure of the imagesensor 113 is known, the ratio may be either calculated or obtained bycomputer simulation. Or the ratio may also be determined by actualmeasurement using a calibration sample.

FIG. 11 illustrates a state where the object 30 is put on the uppersurface 48 of the image sensor 113 and irradiated with light rays comingfrom an irradiation direction that intersects with the imaging surfaceat right angles. Portions of the object 30 which face the respectiveupper surfaces of the subpixels 50 a, 50 b, 50 c, and 50 d will behereinafter referred to as “subpixel portions S1 to S4” of the object30. Also, the optical transmittances of those subpixel portions S1through S4 will be identified herein by S₁ to S₄, too. Thetransmittances S₁ to S₄ of the object 30 depend on the structure ortissue architecture of the object 30 and form part of pixel informationof a high-definition image of the object 30. The amounts of light raystransmitted through the subpixel portions S1 to S4 of the object 30 andincident on the photodiode 40 can be represented by values obtained bymultiplying the transmittances S₁ to S₄ by the ratios described above.If the numerical values defining the ratios are ½, 1, ½ and 0,respectively, the amounts of light rays transmitted through therespective subpixel portions S1 to S4 and then incident on thephotodiode 40 are S₁/2, S₂, S₃/2 and 0, respectively. That is to say,the output of the photodiode 40 will have magnitude corresponding toS₁/2+S₂+S₃/2.

As can be seen, light rays that have been transmitted through therespective subpixel portions S1 to S4 of the object 30 can be incidenton a single photodiode 40, and therefore, pieces of information aboutthose subpixel portions S1 through S4 (their transmittances) will beconvoluted together in the output value of the photodiode 40. That iswhy the set of numerical values such as “½, 1, ½, 0” described above maybe called a “convolution ratio”. Such a set of numerical value may bedealt with as a vector.

By capturing images of the object 30 in multiple different irradiationdirections and by getting output values of the photodiodes 40 so thateach output value becomes an independent set of vectors, of which thenumber is equal to or greater than that of the transmittances S₁ to S₄,the transmittances S₁ to S₄ can be determined by computation.

FIG. 12A is a table showing the relations between the output values A₁to A₄ of the photodiode 40 which have been obtained in the respectiveirradiation directions J1 to J4 and the transmittances S₁ to S₄. The setof output values A₁ to A₄ can be regarded as components of vector A_(j),where j is a number of 1 to 4 specifying one of the irradiationdirections J1 to J4.

In the following description, a matrix consisting of the numericalvalues in four columns and four rows on the table shown in FIG. 12A willbe hereinafter represented by M^(i) _(j), where i is equal to the numberof subpixels allocated to a single pixel. A vector having thetransmittances S₁ to S₄ which are unknown quantities as its componentscan be represented by S_(i). In this case, the equation M^(i)_(j)S_(i)=A_(j) is satisfied.

This equation M^(i) _(j)S_(i)=A_(j) is satisfied with respect to everypixel. Thus, set of simultaneous equations for the entire imagingsurface can be obtained. If the number of pixels is N, then the matrixcan be extended as in the table shown in FIG. 12B. In such a case,M^(ik) _(j)S_(i)=A^(k) _(j) is satisfied, where N is the number ofpixels of the image that is the object of computation, k is a number of1 to N specifying one of the N pixels.

Once the matrix M^(ik) _(j) has been determined, the vector S_(i) can beobtained by calculating the inverse matrix of the matrix M^(ik) _(j)with respect to the vector A^(k) _(j) obtained by capturing images.

According to the method described above, to obtain a high-resolutionimage at a zoom power n, the object may be irradiated with light beamscoming from n² different light source positions, thereby capturing n²images. In this case, calibration needs to be made in advance withrespect to the “convolution ratio” represented as elements of the matrixM^(i) _(j).

According to another method, calibration can be made automatically byincreasing the number of light sources provided.

In general, the numerical values in the matrix M^(i) _(j) (matrixelements) involve some errors. However, if these errors are evaluatedwhen the inverse matrix is calculated to obtain the vector S_(i), asolution which is even closer to the true value can be obtained and thedevice can be calibrated. This point will be described in further detailbelow.

If the object is irradiated with light beams coming from n irradiationdirections (or light sources) where n is equivalent to the zoom power n(as in FIGS. 12A and 12B), the errors that the numerical values includedin the matrix M^(i) _(j) may involve can neither be calculated norcalibrated. However, if the zoom power is n and if the number ofirradiation directions (or light sources) provided is greater than n²(more specifically, equal to or greater than ((n+1)×(n+1)−1)), thenthere is no need to calibrate the numerical values included in thematrix M^(i) _(j) in advance.

To reduce the influence of such errors that the numerical valuesincluded in the matrix M^(i) _(j) involve, the shooting session may becarried out with the number of irradiation directions increased by oneas in the tables shown in FIGS. 13A and 13B. This point will bedescribed as to a situation where the zoom power n is two for the sakeof simplicity. That is to say, in the example to be described below, thenumber of subpixels is two and the object is irradiated withilluminating light beams coming from three different irradiationdirections J1, J2 and J3.

FIG. 14 is a table showing exemplary ratios in such a situation. Theunknown quantities are S₁ and S₂. The pixel value obtained when an imageis captured with an illuminating light beam coming from the irradiationdirection J1 is A₁. The pixel value obtained when an image is capturedwith an illuminating light beam coming from the irradiation direction J2is A₂. And the pixel value obtained when an image is captured with anilluminating light beam coming from the irradiation direction J3 is A₃.Specific numerical values measured by capturing images are substitutedinto A₁, A₂ and A₃.

FIG. 15 shows three simultaneous equations to be satisfied between S₁,S₂, A₁, A₂ and A₃. In these equations, the coefficients for S₁ and S₂are equal to the numerical values of the ratios shown in FIG. 14. Toobtain the two unknown quantities S₁ and S₂, basically there should betwo simultaneous equations. However, by using three simultaneousequations as shown in FIG. 15, even if the numerical values of theratios shown in FIG. 14 involve errors, a solution close to the truevalue can still be obtained.

FIG. 16 is a graph in which three lines represented by the threeequations shown in FIG. 15 are drawn on a two-dimensional coordinateplane, of which the abscissa is S₁ and ordinate is S₂. If the ratiosshown in FIG. 14 had no errors, then the three lines shown in FIG. 16would intersect with each other at a point and the coordinates of thatpoint would correspond to the solution to be obtained. Actually,however, three intersections are observed due to errors in the exemplarygraph shown in FIG. 16. These three intersections should be located inthe vicinity of the true value. Thus, the central point of the deltazone surrounded with these three lines may be selected as the solutionof the three simultaneous equations. The solution may also be obtainedas a point at which the sum of the squares of the distances from thethree lines becomes minimum. The solution thus determined is anestimated value of S₁ and S₂. As shown in FIG. 16, the distances fromthe estimated solution to the three lines may be defined as errors.

According to the present disclosure, an image of the object 30 iscaptured with substantially parallel light rays transmitted through theobject 30 (with a divergent angle of 1/100 radians or less). There is noneed to arrange any imaging lens between the object 30 and the imagesensor 113. And the object 30 may be arranged close to the image sensor113. The interval between the imaging surface of the image sensor 113and the object 30 is typically equal to or shorter than 100 μm and maybe set to be approximately 1 μm, for example.

For example, by irradiating the object with light beams coming from 25different directions, the resolution can be increased as much asfivefold at maximum. Supposing N is an integer which is equal to orgreater than two, if images are captured by irradiating the object withlight beams from N̂2 different directions, the resolution can beincreased N fold at maximum. In 2-dimensional case, to increase theresolution N fold with respect to a certain object 30 means that eachpixel includes N̂2 subpixels. If twenty-five low-resolution images arecaptured by sequentially irradiating the object 30 with light beamscoming from twenty-five light sources which are arranged in five rowsand five columns, each pixel will have subpixels arranged in five rowsand five columns.

In the image forming apparatus of the present disclosure, while multiplelow-resolution images are being captured with the direction of theilluminating light beam changed, it is beneficial that the object 30should not move or be deformed.

As can be seen from the foregoing description, it is helpful to setappropriately the directions of the light beams to irradiate the object30. Also, the object 30 and the image sensor 113 may be surrounded withwalls that shut out external light so that no light other than theilluminating light is incident on the object 30 at least while imagesare being captured.

Embodiments of the present disclosure will now be described in furtherdetail.

Embodiment 1

An image forming apparatus as a first embodiment of the presentdisclosure will be described with reference to FIG. 17, which is a blockdiagram showing an example of configuration for an image formingapparatus according to this embodiment. As shown in FIG. 17, this imageforming apparatus 1 includes an image capturing processing section 11with an illuminating function and an image capturing function, an imageprocessing section 12 which generates and outputs a high-resolutionimage based on low-resolution images obtained by the image capturingprocessing section 11, and a storage device 13 which stores light sourceposition information and the low-resolution image.

The image capturing processing section 11 includes the illumination unit111, the object supporting portion 112 and the image sensor 113. Theillumination unit 111 has the configuration described above, and canirradiate the object with parallel light beams with a predeterminedilluminance (and with a divergent angle of 1/100 radians or less, forexample) from multiple directions. The object supporting portion 112supports the object so that the interval between the imaging surface ofthe image sensor 113 and the object becomes equal to or shorter than 10mm (typically 1 mm or less).

The illumination unit 111 of this embodiment includes LEDs as lightsources for example. The illumination unit 111 may include LEDs in thethree colors of RGB, which are arranged at respective positions.However, the light sources do not have to be LEDs but may also be lightbulbs, laser diodes or fiber lasers as well. When light bulbs are used,a lens or reflective mirror which transforms the light emitted from thelight bulbs into a parallel light beam may be used. Still alternatively,the light sources may also emit infrared light or ultraviolet light.Color filters which either change or filter out the wavelengths of thelight emitted from the light sources may be arranged on the opticalpath. In this embodiment, twenty-five sets of light sources are arrangedat twenty-five different light source positions.

The illumination unit 111 may include either a plurality of lightsources as shown in FIGS. 5A to 5C or a single light source which issupported movably as shown in FIG. 6 so as to change the direction ofthe light that is going to be incident on the object.

The object supporting portion 112 is a member for supporting the objectduring an image capturing session, and may be the upper surface of theimage sensor 113. Optionally, the object supporting portion 112 may havea mechanism to support the object so that its position does not changeduring an image capturing session. The object supporting portion 112 maybe configured to put the object 30 on the image sensor 113 with almostno gap left between them.

FIG. 18 illustrates the relative arrangement of the object 30 on theimage sensor 113 with respect to the light source 10.

The distance D from the light source 10 to the object 30 may be set tobe equal to or longer than 1 m, for example. To prevent the image fromgetting blurred, the interval L between the imaging surface of the imagesensor 113 and the object 30 may be set to be equal to or smaller than100 μm (=1×10⁻⁴ m), e.g., 1 μm (=1×10⁻⁶ m). Supposing D=1 m and L=1×10⁻⁶m, if the light source 10 shifts X m horizontally and laterally, thelight ray going out of the light source 10 and passing through a point Aon the object 30 will be incident at a point on the imaging surfacewhich has also shifted ΔX m. Since ΔX/X=D/L is satisfied, X may bereduced to 0.1 m or less to decrease ΔX to 0.1 μm (=1×10⁻⁷ m) or less.It is easy to set the positional shift X of the light source 10 to be0.1 m (=10 cm) or less when the position of the light source 10 isadjusted. When an image sensor 113 with a pixel pitch K of about 1 μm isused, the distance from the image sensor 113 to the light source 10 maybe set to be approximately 1 m. In that case, even if the light sourcehas caused a positional shift X of several cm or so, the image qualitywill not be debased. Also, in view of these considerations, if red,green and blue light sources (which will be hereinafter referred to as“RGB light sources”) are arranged in a particular irradiation directionclose to each other so as to fall within the range of 0.1 m (=10 cm) orless, those light sources may be handled as a single light source.

In this embodiment, the image sensor 113 may be comprised ofapproximately 4800×3600 pixels, for example. In that case, the pixelpitch K may be set to be approximately 1.3 μm, for example. Also, theinterval between the imaging surface and the upper surface of the imagesensor, i.e., the interval L between the imaging surface and the object,may be set to be approximately 1.3 μm, for example. In this embodiment,the aperture ratio of the image sensor 113 may be, but does not have tobe, 25%.

FIG. 19 shows an exaggerated distribution of the angles of incidence ofmultiple light rays that have been emitted from a single light source10. A light ray is incident perpendicularly onto a region which islocated right under the light source 10. On the other hand, a light rayis incident obliquely onto a region which is located at an end portionof the imaging surface. Suppose the distance D from the imaging surfaceto the light source 10 is set to be approximately 1 m. The distance Cfrom the center of the image sensor to the end portion is at most 10 mm(=1×10⁻² m). Also, in this example, L=1×10⁻⁶ m. Ideally, the lightcoming from the light source should be incident perpendicularly, but isincident obliquely onto such an end portion of the imaging surface. Thatis why the point of incidence of such an obliquely incident light rayshifts ΔX with respect to the point of incidence of the perpendicularlyincident light ray. When the exemplary set of numerical values describedabove is adopted, C/D=Δx/L is satisfied. Thus,Δx=(LC)/D=(1×10⁻⁶×1×10⁻²)/1=1×10⁻⁸=10 nm is satisfied. That is to say,depending on whether the light ray has passed through the center or anend portion of the image sensor (i.e., depending on which portion of theobject the light ray has passed through) before being incident on thephotodiode, a magnitude of shift Δx of at most 10 nm will be caused. Ifthe pixel pitch K is 1 μm (=1×10⁻⁶ m), Δx=10 nm (=1×10⁻⁸ m) is smallerthan the pixel pitch K by two digits. That is why as long as thedistance D from the imaging surface to the light source 10 is set to bean appropriate value with the size of the imaging surface taken intoaccount, the irradiation direction with respect to the object may beregarded as constant for the same light source, no matter where thelight come from with respect to the object.

Now take a look at FIG. 17 again. The image processing section 12 ofthis embodiment includes an illumination condition adjusting section121, an image information getting section 122, an estimate calculatingsection 123, and an image forming processing section 124. Thesecomponents may be implemented as respective functional blocks of acomputer that performs the function of the image processing section 12and may have their functions performed by executing a computer program.The storage device 13 includes a light source position informationserver 131 and a low-resolution image server 132. The storage device 13may be a hard disk drive, a semiconductor memory or an optical storagemedium, or may also be a digital server which is connected to the imageprocessing section 12 through a digital network such as the Internet.

The illumination condition adjusting section 121 of the image processingsection 12 adjusts various illumination conditions (including the lightsource's position, its brightness, the light emission interval, andilluminance) imposed on the illumination unit 111. The image informationgetting section 122 controls the image sensor 113 with the illuminationconditions set appropriately for the illumination unit 111 and makes theimage sensor 113 capture images as the light sources to be turned ON arechanged one after another. The image information getting section 122receives data about the images (low-resolution images) captured by theimage sensor 113 from the image sensor 113. Also, the image informationgetting section 122 gets pieces of information defining the illuminationconditions (including irradiation directions, emission intensities,illuminance and wavelengths) from the illumination condition adjustingsection 121 in association with the image data received.

The light source position information server 131 stores, as a databaseof positions, information about the light source's position provided bythe image information getting section 122. In this example, the lightsource position information server 131 also stores a database of thematrix shown in FIG. 13B. Every time the numerical values (elements) ofthe matrix are adjusted by the estimate calculating section 123 to bedescribed later, this database is rewritten.

The low-resolution image server 132 stores, as an image database, dataabout the low-resolution image gotten through the image informationgetting section 122 and information about the illumination conditionsthat were adopted when the low-resolution image was captured. When theimage forming processing (to be described later) gets done, the dataabout the low-resolution image may be deleted from the image database.

In response to a signal indicating that an image capturing session hasended from the image information getting section 122, the estimatecalculating section 123 of the image processing section 12 respectivelygets light source position information and a low-resolution image fromthe light source position information server 131 and low-resolutionimage server 132 of the storage device 13. Then, the estimatecalculating section 123 makes computations based on the principledescribed above, thereby estimating optical transmittances of subpixelportions constituting a high-resolution image and determining whether ornot the estimation is a proper one. If the estimation is a proper one,the estimate calculating section 123 outputs the high-resolution image.Otherwise, the estimate calculating section 123 changes the light sourceposition information. In performing the estimation operation, theestimate calculating section 123 makes reference to the database in thelight source position information server 131, gets the ratio definingthe numerical values (elements) in the matrix, and forms ahigh-resolution image based on the output of the image sensor 113. Inthis case, an estimated value and an error are obtained as describedabove. If the error exceeds a reference value (of 5%, for example), theprocessing of correcting the numerical value representing the ratio intoanother value may be performed. The error ratio may be represented by((error)/|(S₁, S₂, . . . , S₂₅)|×100), for example.

Using this error, the experimental system may also be calibrated at thesame time in order to increase the accuracy of the experiment from thenext time and on. For example, if the estimated value and three linesdrawn are as shown in FIG. 16, calibration can be made and the error canbe eliminated by translating and correcting the three lines so that allof those three lines pass through the same estimated value.

Next, inverse matrix is calculated with the error rated. The conditionalexpression to use for that purpose generally tends to be a verycomplicated one. However, in calculating an estimated value concerningthe pixels that form a rectangular block of a high-resolution image,only pixels falling within limited range of the rectangular block willaffect significantly. For example, if calculations are made on the basisof a rectangular block comprised of 12×12 pixels as shown in FIG. 20,the calculations can be done at high speeds with the calculation costcut down.

If the estimate calculating section 123 needs to get those calculationsdone over the entire image capturing area, then an opaque portion may beprovided to prevent light from entering the image capturing area fromoutside of the object range. To prevent light from entering the imagecapturing area from a region where the object is not present, an opaquearea 404 which limits the image capturing range may be arranged on theobject supporting portion as shown in FIG. 21. Alternatively, an opaquemember 403 may be arranged on a side surface of the image sensor 113 asshown in FIG. 22.

The image forming processing section 124 composes a high-resolutionimage based on the image information which has been provided by theestimate calculating section 123 and of which the properness has beenproved, and subjects the image to color correction, de-mosaicing (alsocalled de-pixelization) processing, grayscale correction (γ correction),YC separation processing, overlap correction and other kinds ofcorrection. The high-resolution image thus obtained is presented on adisplay (not shown) or output to a device outside of the image formingapparatus 1 through an output section. The high-resolution image outputthrough the output section may be written on a storage medium (notshown) or presented on another display.

The low-resolution image server 132 stores data about the low-resolutionimage which has been gotten by the image information getting section122. While the estimate calculating section 123 is composing an image,necessary low-resolution image data is retrieved from the database ofthis low-resolution image server 132. When the image forming processinggets done, unnecessary data may be deleted from the low-resolution imageserver 132.

According to this embodiment, if the number of irradiation directions isset to be equal to or greater than ((n+1)×(n+1)−1), a high-resolutionimage composed of n×n pixels can be obtained. Thus, the image formingapparatus of the present disclosure can obtain an image which has beenzoomed in at a high zoom power over the entire area without using amicroscope which usually needs a lot of time for focusing. Consequently,even if the object is a pathological sample with a microscopic tissue,image data can be obtained at a high zoom power in a short time.

Next, it will be described with reference to FIG. 23 how the imageforming apparatus 1 of the embodiment described above operates. FIG. 23is an exemplary flowchart showing the procedure in which the imageforming apparatus 1 gets an image.

In FIG. 23, first of all, the object is mounted on or above the objectsupporting portion 112 (in Step S201). In this example, the object is apathological specimen. However, the object may also belight-transmitting sample, of which the thickness is about several μmand of which the shape does not change during the image capturingsession (such as a cell or a sliced tissue). Optionally, the imagecapturing session may be carried out with slide glass reversed. In thiscase, cover glass 32 may be put on the upper surface of the image sensorand the sample may be put on the cover glass. The slide glass may bearranged over the image sensor.

Next, to get low-resolution images, images are captured with twenty-fivelight sources sequentially turned ON one after another. For example, bydefining i=1 (in Step S202), only the i^(th) light source is turned ON(in Step S203). An image is captured with the contrast ratio adjusted(in Step S204). And data about the image captured is stored in the imagebuffer in the storage device (in Step S205).

Next, i is defined to be i+1 (in Step S206) and then decision is madewhether or not i has exceeded N=25 which is the number of images to becaptured (in Step S207). Images are captured over and over again until iexceeds 25.

If the decision has been made that i>N is satisfied (i.e., if the answerto the query of the processing step S207 is YES), the image formingprocessing is carried out. For example, a pixel conditional expressionto be used to form, by calculation, a high-resolution image based onrespective low-resolution images is defined (in Step S208) and a pixelestimation calculation is carried out (in Step S209). Next, the error israted (in Step S210). If the error has turned out to be less than areference value, the high-resolution image obtained is output (in StepS212). On the other hand, if the error has turned out to be equal to orgreater than the reference value, the numerical values stored in thestorage device are corrected and then a pixel conditional expression isredefined (in Step S208).

According to the exemplary flow described above, a single low-resolutionimage is supposed to be captured for each light source position for thesake of simplicity. However, this is only an example of an embodiment ofthe present disclosure. If three LED light sources (e.g., RGB LED lightsources) are arranged at each light source position, then threelow-resolution images (RGB images) may be captured for each light sourceposition. Consequently, by capturing low-resolution images in fullcolors, a full-color high-resolution image can be obtained eventually.

In the example described above, the number of multiple differentirradiation directions is supposed to be twenty-five. However, thenumber of irradiation directions may also be less than or greater thantwenty-five.

Embodiment 2

FIG. 24 is an example of block diagram illustrating an image formingapparatus as a second embodiment of the present disclosure. In the imageforming apparatus 1 of this embodiment, its image processing section 12further includes a light source position determining section 125 whichcalibrates the position of the light source, which is a difference fromthe image forming apparatus of the first embodiment described above.Thus, according to this embodiment, the light source position can beadjusted.

Next, an operation unique to this embodiment will be described withreference to FIG. 25.

If the decision has not been made in Step S210 that the error is lessthan the reference value (i.e., if the answer to the query of theprocessing step S210 is NO), then the light source position informationinvolving the most significant error is corrected (in Step S211). Then,if necessary, another low-resolution image is captured all over againwith the object irradiated with light coming from the adjusted lightsource position (in Step S203).

Embodiment 3

According to a third embodiment of the present disclosure, the vectorS_(j) representing a high-resolution image is not obtained bycalculating the inverse matrix. Instead, a high-resolution image isgenerated by using general super-resolution processing. To increase theresolution using an inverse matrix, supposing the image size of ahigh-resolution image is w×h, the inverse matrix of a matrix wh×whshould be obtained. That is why the larger the image size, the moredifficult it is to get computation done. On the other hand,super-resolution processing can get done by making computation withinreal time. For that reason, it is easy even for a computer with lowcomputational ability to get the super-resolution processing done, whichis beneficial.

According to the super-resolution processing of this embodiment,computation is carried out in a frequency domain using a Wiener filterrepresented by the following Equation (1):

$\begin{matrix}{{H(Y)} = {{H(D)}^{- 1}{H(X)}}} & (1) \\{{H(D)}^{- 1} = \frac{H(D)}{{H(D)}^{2} + \Gamma}} & (2)\end{matrix}$

where Y is a zoomed-in image to be described below, X is ahigh-resolution image to be obtained, D is the convolution ratio, andH(.) represents conversion into a frequency domain. H(D)⁻¹ is given byEquation (2). In the denominator of the right side of Equation (2), Γ isa parameter representing an SNR.

An example of Y will be described with reference to FIG. 26, whichillustrates how to form a zoomed-in image 2801 to obtain ahigh-resolution image at a zoom power of 2×. Y corresponds to thezoomed-in image 2801.

To obtain the zoomed-in image 2801 at a zoom power of 2×, the object isirradiated with light beams from four directions. In FIG. 26, thelow-resolution image 2802 is captured when the object is irradiated witha light beam coming from right over the object. The low-resolution image2803 is captured when the light source is moved by a different distanceonly in the X-axis direction. The low-resolution image 2804 is capturedwhen the light source is moved by a different distance only in theY-axis direction. And the low-resolution image 2805 is captured when thelight source is moved along the bisector between the X- and Y-axisdirections.

As can be seen, according to this embodiment, super-resolutionprocessing is carried out by making computations in a frequency domainusing a Wiener filter. However, this is only an example and thesuper-resolution processing does not always have to be carried out inthis manner. For example, the super-resolution processing may also becarried out using the following update equations (3) and (4):

$\begin{matrix}{E = \left( {Y - {D*X}} \right)^{2}} & (3) \\{x_{i,j}^{t + 1} = {x_{i,j}^{t} - {\lambda \frac{\partial E}{\partial x_{i,j}}}}} & (4)\end{matrix}$

Equation (3) is obtained by differentiating Equation (4) with X_(i,j)^(t). In Equation (4), X_(i,j) ^(t) represents the (i, j)^(th) pixelvalue in the image X when the same computation is carried out for thet^(th) time, and λ represents a parameter at the time of update.Optionally, an L2 norm cost function or L1 norm cost function may alsobe used to Equation (3) with noise in the image taken into account.

According to this embodiment, a high-resolution image can be obtainedwithin real time. In addition, just like when the resolution isincreased using an inverse matrix, the super-resolution processing canalso be carried out based on pixels falling within only a limited rangeas shown in FIG. 20. On top of that, the image may be divided intomultiple small regions and different kinds of resolution enhancementprocessing, including inverse matrix computation, can be carried out inthe respective small regions. In that case, in regions where theresolution enhancement processing does not need to be carried out (as insmall regions where the object is not shot on the image), no resolutionenhancement processing may be carried out. The speed and accuracy ofcomputation vary depending on the method in obtaining a high-resolutionimage. That is why by getting computation done speedily in anunimportant region around the background and by carrying out theprocessing accurately in a small region where the object is shot on theimage, only a region that the user wants to view can have its resolutionincreased selectively.

MODIFIED EXAMPLE

FIG. 27 schematically illustrates a configuration for a modified exampleincluding a holder which holds the object and image sensor (which willbe hereinafter referred to as an “object of shooting 140”) in anattachable and removable state. The object of shooting 140 can be aprepared specimen in which the object and image sensor are combinedtogether. In this example, an angle of illumination adjusting sectionhas a mechanism which changes the orientation of the object of shooting140. This mechanism includes two gonio systems 120 which can rotate theobject within a perpendicular planes that intersect at right angles. Thecenter of rotation 150 of the gonio systems 120 is located at the centerof the object in the object of shooting 140. In this modified example,the gonio system 120 can change the irradiation direction of theilluminating light, and therefore, the light source 10P may be fixed.Also, in this modified example, the light source 10P is configured toemit a parallel light beam. Alternatively, the image forming apparatusof this modified example may include a plurality of parallel light beamsources 10P as shown in FIG. 28.

In this case, the configuration in which the light source 10P is fixedand the object of shooting 140 is moved is advantageous than theconfiguration in which the object of shooting 140 is fixed and the lightsource 10P is moved, because the former configuration contributes togetting the shooting session done in a shorter time. This is alsobecause the distance L1 from the object of shooting 140 to the lightsource 10P is so much longer than the interval L2 between the object andthe image sensor that form the object of shooting 140 that the lightsource 10P should be significantly moved proportionally to the longdistance according to the latter configuration. By getting the shootingsession done in a shorter time, even if the object changes with time ona second basis (e.g., even if the object is luminescence from abiological sample), an appropriate image can also be shot.

FIG. 29 illustrates a configuration for a modified example in which amechanism for changing the object's orientation includes a gonio system120 and a rotating mechanism 122. By combining the rotation of theobject of shooting 140 which is caused by the gonio system 120 within aperpendicular plane with the rotation of the object of shooting 140which is caused by the rotating mechanism 122 around a perpendicularaxis, the object of shooting 140 can be irradiated with an illuminatinglight beam coming from any arbitrary irradiation direction. A point 150is located at the center of rotation of the gonio system 120 and at thecenter of rotation of the rotating mechanism 122. Alternatively, theimage forming apparatus of this modified example may include a pluralityof parallel light beam sources 10P as shown in FIG. 30.

FIG. 31 illustrates an exemplary optical system which can increase thedegree of parallelism of the light emitted from a light source and canmake a parallel light beam incident on the object. In this example, alens 130 which collimates divergent light emitted from the light sourceis provided for an XY moving mechanism (moving stage) 124. Along withthe moving stage 124, the object of shooting 140 can be moved by anarbitrary distance along the X axis and/or the Y axis within ahorizontal plane.

FIG. 32 illustrates how an illuminating light beam is incident obliquelyonto the object of shooting 140 which has moved a predetermined distancein a specified direction within a horizontal plane. Even if the positionof the light source 10 a is fixed, the irradiation direction of theilluminating light beam can also be controlled by adjusting the positionof the object of shooting 140. Alternatively, the image formingapparatus of this modified example may include a plurality of lightsources as shown in FIG. 33. If a plurality of light sources 10 a, 10 band 10 c are provided as shown in FIG. 33, then the mechanism to movethe object of shooting 140 may be omitted or an XY moving mechanism(moving stage) 124 may be provided as shown in FIG. 34. By changing theposition of the light source 10 a, 10 b, 10 c to turn ON and/or theposition of the object of shooting 140 as shown in FIGS. 36 and 37, anilluminating light beam can be made incident on the object of shooting140 at any intended angle of incidence.

FIG. 38 illustrates schematically a configuration for a modified examplein which two gonio systems 120 support a parallel light beam source 10P.FIG. 39 illustrates schematically a configuration for a modified examplein which a gonio system 120 and a rotating mechanism 122 support aparallel light beam source 10P.

In these modified examples, either a light source which emits a parallellight beam or an optical system which increases the degree ofparallelism of the light emitted from a light source is used. However,these are just examples of embodiments of the present disclosure. Asdescribed above, if the distance from the light source to the object issufficiently long, light which can be regarded as a substantiallyparallel light beam will be incident on the object.

In an embodiment of the present disclosure, within the imaging surface,the vertical size of each photodiode is expressed as 1/s of that of itsassociated pixel region, and the horizontal size of that photodiode isexpressed as 1/t of that of the pixel region, where s and t are bothreal numbers and do not have to be integers. The aperture ratio is givenby (1/s)×(1/t). In an embodiment of the present disclosure, the objectmay be irradiated with illuminating light beams in respectiveirradiation directions in which a different angles of irradiation aredefined with respect to the vertical direction in the imaging surface,and images may be captured in those directions, respectively. Also, theobject may be irradiated with illuminating light beams in respectiveirradiation directions in which b different angles of irradiation aredefined with respect to the horizontal direction in the imaging surface,and images may be captured in those directions, respectively. In thiscase, a and b are integers which satisfy a≧s and b≧t. According to anembodiment of the present disclosure, low-resolution images are capturedon a “a×b” basis, and an image, of which the resolution has increased“a×b” fold, can be obtained based on these low-resolution images. Itshould be noted that the product of (1/s)×(1/t) representing the imagesensor's aperture ratio and a×b becomes equal to or greater than one.

An image forming apparatus according to the present disclosure mayinclude an illumination system with a tilting mechanism which tilts theobject and the image sensor together. In that case, even if the lightsource's position is fixed, the irradiation direction with respect tothe object can be changed by getting the object and the image sensorrotated by the tilting mechanism. Such an illumination system cansequentially emit illuminating light beams from multiple differentirradiation directions with respect to the object by tilting the objectand the image sensor together.

An image forming method according to an aspect of the present disclosureincludes the steps of: sequentially emitting illuminating light beamsfrom multiple different irradiation directions with respect to an objectand irradiating the object with the illuminating light beams; capturinga plurality of different images in the multiple different irradiationdirections, respectively, using an imaging device which is arranged at aposition where the illuminating light beams that have been transmittedthrough the object are incident; and forming a high-resolution image ofthe object, having a higher resolution than any of the plurality ofimages, based on the plurality of images.

Also, an image forming apparatus according to the present disclosure mayinclude the illumination unit and image sensor described above and ageneral-purpose computer. The computer may be set up to: make theillumination unit sequentially emit illuminating light beams frommultiple different irradiation directions with respect to an object andirradiate the object with the illuminating light beams; capture aplurality of different images in the multiple different irradiationdirections, respectively, using an imaging device which is arranged at aposition where the illuminating light beams that have been transmittedthrough the object are incident; and form a high-resolution image of theobject, having a higher resolution than any of the plurality of images,based on the plurality of images. Such an operation may be performed inaccordance with a computer program which is stored on a storage medium.

If a light source which irradiates the object with light and of whichthe orientation and position are fixed is used and if a tiltingmechanism which tilts the object at multiple tilt angles is provided, animage sensor which is arranged at a position where the light that hasbeen transmitted through the object is incident and the object can gettilted together by the tilting mechanism, and a plurality of images canbe captured at the multiple tilt angles.

An image forming apparatus as one implementation of the presentdisclosure comprises: an illumination system which sequentially emitsilluminating light beams from multiple different irradiation directionswith respect to an object and irradiates the object with theilluminating light beams; an image sensor which is arranged at aposition where the illuminating light beams that have been transmittedthrough the object are incident and which captures a plurality ofdifferent images in the multiple different irradiation directions,respectively; an image processing section which forms a high-resolutionimage of the object, having a higher resolution than any of theplurality of images, based on the plurality of images; and a memorywhich stores data about the ratio of light rays that have been incidenton a photoelectric conversion section of each pixel of the image sensorto light rays that have passed through the upper surface of a pluralityof subpixels included in the pixel with respect to each of the multipleirradiation directions, wherein the image processing section forms thehigh-resolution image of the object based on the data that has beenretrieved from the memory by extracting, as a vector, a set of pixelvalues associated with the multiple irradiation directions from pixelvalues that form each of the plurality of images.

In one embodiment, the image processing section forms thehigh-resolution image of the object by multiplying the inverse matrix ofa matrix having the ratio as its element by the vector of the pixelvalues based on the data that has been retrieved from the memory byextracting, as a vector, a set of pixel values associated with themultiple irradiation directions from pixel values that form each of theplurality of images.

In one embodiment, the image processing section forms thehigh-resolution image of the object by performing super-resolutionprocessing using, as a vector, a set of pixel values that are associatedwith the multiple irradiation directions and that have been extractedfrom pixel values that form each of the plurality of images.

In one embodiment, the object is arranged close to the image sensor, andthe object and the image sensor face each other with no lensesinterposed between them.

In one embodiment, the interval between the image sensor's imagingsurface and the object is equal to or shorter than 100 μm.

In one embodiment, each of the plurality of images includes imagesrepresenting respectively different portions of the object.

In one embodiment, the object is fixed onto the image sensor, and theapparatus includes a holder which holds the image sensor in anattachable and removable state.

In one embodiment, supposing a and b are integers which are equal to orgreater than two, the illuminating light beams are made to be incidenton the object from different irradiation directions so that a differentangles of irradiation are defined with respect to a vertical directionwithin the imaging surface of the image sensor, and images are capturedin those irradiation directions, the illuminating light beams are madeto be incident on the object from different irradiation directions sothat b different angles of irradiation are defined with respect to ahorizontal direction within the imaging surface, and images are capturedin those irradiation directions, and the product of the aperture ratioof the image sensor and a×b becomes equal to or greater than one.

In one embodiment, the illumination system is able to emit light beamsfalling within respectively different wavelength ranges.

In one embodiment, the illumination system includes a light source whichsequentially moves to multiple different positions corresponding to themultiple different irradiation directions and emits the illuminatinglight beams from those positions one after another.

In one embodiment, the illumination system includes a plurality of lightsources which are arranged at multiple different positions correspondingto the multiple different irradiation directions and emit theilluminating light beams sequentially.

In one embodiment, the illumination system includes a tilting mechanismwhich tilts the object and the image sensor together, and by tilting theobject and the image sensor together, the illumination systemsequentially emits illuminating light beams from multiple differentirradiation directions with respect to the object and irradiates theobject with the illuminating light beams.

In one embodiment, the illumination system includes a mechanism whichchanges at least one of the object's orientation and position.

In one embodiment, the mechanism includes at least one of a gonio systemand a moving stage.

In one embodiment, the mechanism includes an optical system whichincreases the degree of parallelism of the illuminating light beam.

In one embodiment, the illumination system includes an optical systemwhich increases the degree of parallelism of the illuminating lightbeam.

An image forming method as another implementation of the presentdisclosure comprises the steps of: sequentially emitting illuminatinglight beams from multiple different irradiation directions with respectto an object and irradiating the object with the illuminating lightbeams; capturing a plurality of different images in the multipledifferent irradiation directions, respectively, using an imaging devicewhich is arranged at a position where the illuminating light beams thathave been transmitted through the object are incident; forming ahigh-resolution image of the object, having a higher resolution than anyof the plurality of images, based on the plurality of images; andstoring in a memory data about the ratio of light rays that have beenincident on a photoelectric conversion section of each pixel of theimage sensor to light rays that have passed through the upper surface ofa plurality of subpixels included in the pixel with respect to each ofthe multiple irradiation directions, wherein the step of forming ahigh-resolution image of the object includes the step of forming thehigh-resolution image of the object by multiplying the inverse matrix ofa matrix having the ratio as its element by the vector of the pixelvalues based on the data that has been retrieved from the memory byextracting, as a vector, a set of pixel values associated with themultiple irradiation directions from pixel values that form each of theplurality of images.

An image forming apparatus as still another implementation of thepresent disclosure comprises an illumination unit, an image sensor and acomputer, wherein the computer is operative to: make the illuminationunit sequentially emit illuminating light beams from multiple differentirradiation directions with respect to an object and irradiate theobject with the illuminating light beams; capture a plurality ofdifferent images in the multiple different irradiation directions,respectively, using the image sensor which is arranged at a positionwhere the illuminating light beams that have been transmitted throughthe object are incident; form a high-resolution image of the object,having a higher resolution than any of the plurality of images, based onthe plurality of images; and store in a memory data about the ratio oflight rays that have been incident on a photoelectric conversion sectionof each pixel of the image sensor to light rays that have passed throughthe upper surface of a plurality of subpixels included in the pixel withrespect to each of the multiple irradiation directions, wherein thecomputer is designed to form the high-resolution image of the object bymultiplying the inverse matrix of a matrix having the ratio as itselement by the vector of the pixel values based on the data that hasbeen retrieved from the memory by extracting, as a vector, a set ofpixel values associated with the multiple irradiation directions frompixel values that form each of the plurality of images.

Yet another implementation of the present disclosure is a program to beused by an image forming apparatus including an illumination unit, animage sensor and a computer, wherein the program is set up to: make theillumination unit sequentially emit illuminating light beams frommultiple different irradiation directions with respect to an object andirradiate the object with the illuminating light beams; capture aplurality of different images in the multiple different irradiationdirections, respectively, using the image sensor which is arranged at aposition where the illuminating light beams that have been transmittedthrough the object are incident; form a high-resolution image of theobject, having a higher resolution than any of the plurality of images,based on the plurality of images; and store in a memory data about theratio of light rays that have been incident on a photoelectricconversion section of each pixel of the image sensor to light rays thathave passed through the upper surface of a plurality of subpixelsincluded in the pixel with respect to each of the multiple irradiationdirections, wherein the program is set up to form the high-resolutionimage of the object by multiplying the inverse matrix of a matrix havingthe ratio as its element by the vector of the pixel values based on thedata that has been retrieved from the memory by extracting, as a vector,a set of pixel values associated with the multiple irradiationdirections from pixel values that form each of the plurality of images.

An image forming apparatus as yet another implementation of the presentdisclosure comprises: an illumination system which sequentially emitsilluminating light beams from multiple different irradiation directions,of which the number is larger than n² (where n is an integer that isequal to or greater than two), with respect to an object and irradiatesthe object with the illuminating light beams; an image sensor which isarranged at a position where the illuminating light beams that have beentransmitted through the object are incident and which captures aplurality of different images in the multiple different irradiationdirections, respectively; an image processing section which forms ahigh-resolution image of the object, of which the resolution is n timesas high as that of any of the plurality of images, based on theplurality of images; and a memory which stores data about the ratio oflight rays that have been incident on a photoelectric conversion sectionof each pixel of the image sensor to light rays that have passed throughthe upper surface of a plurality of subpixels included in the pixel withrespect to each of the multiple irradiation directions, wherein theimage processing section forms the high-resolution image of the objectby multiplying the inverse matrix of a matrix having the ratio as itselement by the vector of the pixel values based on the data that hasbeen retrieved from the memory by extracting, as a vector, a set ofpixel values associated with the multiple irradiation directions frompixel values that form each of the plurality of images.

In one embodiment, every time the high-resolution image is formed, theimage forming apparatus is able to calibrate automatically, withoutusing a mosaic color filter for calibration, the data about the ratio oflight rays to be incident on the photoelectric conversion section ofeach pixel of the image sensor to light rays that have passed throughthe upper surface of a plurality of subpixels included in the pixel withrespect to each of the multiple irradiation directions.

In one embodiment, the illumination system is able to emit light beamsfalling within respectively different wavelength ranges.

In one embodiment, the illumination system includes a light source whichsequentially moves to multiple different positions corresponding to themultiple different irradiation directions and emits the illuminatinglight beams from those positions one after another.

In one embodiment, the illumination system includes a plurality of lightsources which are arranged at multiple different positions correspondingto the multiple different irradiation directions and emit theilluminating light beams sequentially.

In one embodiment, the illumination system includes a tilting mechanismwhich tilts the object and the image sensor together, and by tilting theobject and the image sensor together, the illumination systemsequentially emits illuminating light beams from multiple differentirradiation directions with respect to the object and irradiates theobject with the illuminating light beams.

An image forming method as yet another implementation of the presentdisclosure comprises: sequentially emitting illuminating light beamsfrom multiple different irradiation directions, of which the number islarger than n² (where n is an integer that is equal to or greater thantwo), with respect to an object and irradiating the object with theilluminating light beams; capturing a plurality of different images inthe multiple different irradiation directions, respectively, using animage sensor which is arranged at a position where the illuminatinglight beams that have been transmitted through the object are incident;forming a high-resolution image of the object, of which the resolutionis n times as high as that of any of the plurality of images, based onthe plurality of images; and storing in a memory data about the ratio oflight rays that have been incident on a photoelectric conversion sectionof each pixel of the image sensor to light rays that have passed throughthe upper surface of a plurality of subpixels included in the pixel withrespect to each of the multiple irradiation directions, wherein themethod is designed to form the high-resolution image of the object bymultiplying the inverse matrix of a matrix having the ratio as itselement by the vector of the pixel values based on the data that hasbeen retrieved from the memory by extracting, as a vector, a set ofpixel values associated with the multiple irradiation directions frompixel values that form each of the plurality of images.

An image forming apparatus as yet another implementation of the presentdisclosure comprises an illumination unit, an image sensor and acomputer, wherein the computer is operative to: make the illuminationunit sequentially emit illuminating light beams from multiple differentirradiation directions, of which the number is larger than n² (where nis an integer that is equal to or greater than two), with respect to anobject and irradiate the object with the illuminating light beams;capture a plurality of different images in the multiple differentirradiation directions, respectively, using the image sensor which isarranged at a position where the illuminating light beams that have beentransmitted through the object are incident; form a high-resolutionimage of the object, of which the resolution is n times as high as thatof any of the plurality of images, based on the plurality of images; andstore in a memory data about the ratio of light rays that have beenincident on a photoelectric conversion section of each pixel of theimage sensor to light rays that have passed through the upper surface ofa plurality of subpixels included in the pixel with respect to each ofthe multiple irradiation directions, wherein the computer is designed toform the high-resolution image of the object by multiplying the inversematrix of a matrix having the ratio as its element by the vector of thepixel values based on the data that has been retrieved from the memoryby extracting, as a vector, a set of pixel values associated with themultiple irradiation directions from pixel values that form each of theplurality of images.

Yet another implementation of the present disclosure is a program to beused by an image forming apparatus including an illumination unit, animage sensor and a computer, wherein the program is set up to: make theillumination unit sequentially emit illuminating light beams frommultiple different irradiation directions, of which the number is largerthan n² (where n is an integer that is equal to or greater than two),with respect to an object and irradiate the object with the illuminatinglight beams; capture a plurality of different images in the multipledifferent irradiation directions, respectively, using the image sensorwhich is arranged at a position where the illuminating light beams thathave been transmitted through the object are incident; form ahigh-resolution image of the object, of which the resolution is n timesas high as that of any of the plurality of images, based on theplurality of images; and store in a memory data about the ratio of lightrays that have been incident on a photoelectric conversion section ofeach pixel of the image sensor to light rays that have passed throughthe upper surface of a plurality of subpixels included in the pixel withrespect to each of the multiple irradiation directions, wherein theprogram is set up to form the high-resolution image of the object bymultiplying the inverse matrix of a matrix having the ratio as itselement by the vector of the pixel values based on the data that hasbeen retrieved from the memory by extracting, as a vector, a set ofpixel values associated with the multiple irradiation directions frompixel values that form each of the plurality of images.

An image forming apparatus as yet another implementation of the presentdisclosure comprises: a light source which irradiates an object withlight and of which the orientation and position are fixed; a tiltingmechanism which tilts the object at multiple tilt angles; an imagesensor which is arranged at a position where the light that has beentransmitted through the object is incident, gets tilted along with theobject by the tilting mechanism, and captures a plurality of differentimages at the multiple tilt angles; an image processing section whichforms a high-resolution image of the object, having a higher resolutionthan any of the plurality of images, based on the plurality of images;and a memory which stores data about the ratio of light rays that havebeen incident on a photoelectric conversion section of each pixel of theimage sensor to light rays that have passed through the upper surface ofa plurality of subpixels included in the pixel with respect to each ofthe multiple tilt angles, wherein the image processing section forms thehigh-resolution image of the object based on the data that has beenretrieved from the memory by extracting, as a vector, a set of pixelvalues associated with the multiple tilt angles from pixel values thatform each of the plurality of images.

In one embodiment, the object is arranged close to the image sensor, andthe object and the image sensor face each other with no lensesinterposed between the object and the image sensor.

In one embodiment, the interval between an imaging surface of the imagesensor and the object is equal to or shorter than 100 μm.

In one embodiment, each of the plurality of images includes imagesrepresenting respectively different portions of the object.

In one embodiment, the object is fixed to the image sensor, and theimage forming apparatus includes a holder which holds the image sensorin an attachable and removable state.

In one embodiment, the tilting mechanism includes at least one of agonio system and a moving stage.

In one embodiment, the tilting mechanism includes an optical systemwhich increases the degree of parallelism of the light.

In one embodiment, the light source includes an optical system whichincreases the degree of parallelism of the light.

An image forming method as yet another implementation of the presentdisclosure comprises: irradiating an object with illuminating lightwhich has been emitted from a fixed light source while tilting theobject at multiple tilt angles; capturing a plurality of differentimages at the multiple tilt angles using an image sensor which isarranged at a position where the illuminating light that has beentransmitted through the object is incident; and forming ahigh-resolution image of the object, having a higher resolution than anyof the plurality of images, by synthesizing the plurality of imagestogether, wherein the high-resolution image of the object is formedbased on data about the ratio of light rays that have been incident on aphotoelectric conversion section of each pixel of the image sensor tolight rays that have passed through the upper surface of a plurality ofsubpixels included in the pixel with respect to each of the multipletilt angles.

Yet another implementation of the present disclosure is a programdesigned to make a computer perform: irradiating an object withilluminating light which has been emitted from a fixed light sourcewhile tilting the object at multiple tilt angles; capturing a pluralityof different images at a multiple different irradiation directions usingan image sensor which is arranged at a position where the illuminatinglight that has been transmitted through the object is incident; andforming a high-resolution image of the object, having a higherresolution than any of the plurality of images, by synthesizing theplurality of images together, wherein the high-resolution image of theobject is formed based on data about the ratio of light rays that havebeen incident on a photoelectric conversion section of each pixel of theimage sensor to light rays that have passed through the upper surface ofa plurality of subpixels included in the pixel with respect to each ofthe multiple tilt angles.

Yet another implementation of the present disclosure is an image sensorfor use in an image forming apparatus, the apparatus comprises: anillumination system which sequentially emits illuminating light beamsfrom multiple different irradiation directions with respect to an objectand irradiates the object with the illuminating light beams; a holderwhich holds the image sensor in an attachable and removable state; animage processing section which forms, based on a plurality of differentimages that have been captured by the image sensor in the multipledifferent irradiation directions, respectively, a high-resolution imageof the object having a higher resolution than any of the plurality ofimages; and a memory which stores data about the ratio of light raysthat have been incident on a photoelectric conversion section of eachpixel of the image sensor to light rays that have passed through theupper surface of a plurality of subpixels included in the pixel withrespect to each of the multiple irradiation directions, wherein theimage processing section forms the high-resolution image of the objectbased on the data that has been retrieved from the memory by extracting,as a vector, a set of pixel values associated with the multipleirradiation directions from pixel values that form each of the pluralityof images, the image sensor is arranged so as to be attachable to, andremovable from, the image forming apparatus, the imaging surface of theimage sensor has an object supporting portion which is a region on whichthe object is able to be mounted, and the image sensor is arranged at aposition where the illuminating light beams transmitted through theobject are incident while being held by the holder onto the imageforming apparatus and captures the plurality of different images in themultiple different irradiation directions.

In one embodiment, the image sensor is arranged on slide glass, and heldby the holder onto the image forming apparatus so as to be attachableto, and removable from, the apparatus via a portion of the slide glass.

In one embodiment, an opaque member is arranged on a side surface of theimage sensor.

In one embodiment, an opaque region which limits an image capturingrange is arranged on the object supporting portion.

An image forming apparatus, image forming method, image processingprogram and image sensor according to the present disclosure contributesto getting a high-resolution image with the trouble of adjusting thefocus saved.

While the present invention has been described with respect to exemplaryembodiments thereof, it will be apparent to those skilled in the artthat the disclosed invention may be modified in numerous ways and mayassume many embodiments other than those specifically described above.Accordingly, it is intended by the appended claims to cover allmodifications of the invention that fall within the true spirit andscope of the invention.

What is claimed is:
 1. An image forming apparatus comprising: anillumination system which sequentially emits illuminating light beamsfrom multiple different irradiation directions with respect to an objectand irradiates the object with the illuminating light beams; an imagesensor which is arranged at a position where the illuminating lightbeams that have been transmitted through the object are incident andwhich captures a plurality of different images in the multiple differentirradiation directions, respectively; an image processing section whichforms a high-resolution image of the object, having a higher resolutionthan any of the plurality of images, based on the plurality of images;and a memory which stores data about the ratio of light rays that havebeen incident on a photoelectric conversion section of each pixel of theimage sensor to light rays that have passed through the upper surface ofa plurality of subpixels included in the pixel with respect to each ofthe multiple irradiation directions, wherein the image processingsection forms the high-resolution image of the object based on the datathat has been retrieved from the memory by extracting, as a vector, aset of pixel values associated with the multiple irradiation directionsfrom pixel values that form each of the plurality of images.
 2. Theimage forming apparatus of claim 1, wherein the image processing sectionforms the high-resolution image of the object by multiplying the inversematrix of a matrix having the ratio as its element by the vector of thepixel values based on the data that has been retrieved from the memoryby extracting, as a vector, a set of pixel values associated with themultiple irradiation directions from pixel values that form each of theplurality of images.
 3. The image forming apparatus of claim 1, whereinthe image processing section forms the high-resolution image of theobject by performing super-resolution processing using, as a vector, aset of pixel values that are associated with the multiple irradiationdirections and that have been extracted from pixel values that form eachof the plurality of images.
 4. The image forming apparatus of claim 1,wherein the object is arranged close to the image sensor, and the objectand the image sensor face each other with no lenses interposed betweenthe object and the image sensor.
 5. The image forming apparatus of claim4, wherein the interval between an imaging surface of the image sensorand the object is equal to or shorter than 100 μm.
 6. The image formingapparatus of claim 1, wherein each of the plurality of images includesimages representing respectively different portions of the object. 7.The image forming apparatus of claim 1, wherein the object is fixed tothe image sensor, and the image forming apparatus includes a holderwhich holds the image sensor in an attachable and removable state. 8.The image forming apparatus of claim 1, wherein the illumination systemis able to emit light beams falling within respectively differentwavelength ranges.
 9. The image forming apparatus of claim 1, whereinthe illumination system includes a light source which sequentially movesto multiple different positions corresponding to the multiple differentirradiation directions and emits the illuminating light beams from thosepositions one after another.
 10. The image forming apparatus of claim 1,wherein the illumination system includes a plurality of light sourceswhich are arranged at multiple different positions corresponding to themultiple different irradiation directions and emit the illuminatinglight beams sequentially.
 11. The image forming apparatus of claim 1,wherein the illumination system includes a tilting mechanism which tiltsthe object and the image sensor together, and by tilting the object andthe image sensor together, the illumination system sequentially emitsilluminating light beams from multiple different irradiation directionswith respect to the object and irradiates the object with theilluminating light beams.
 12. The image forming apparatus of claim 11,wherein the illumination system includes a mechanism which changes atleast one of the object's orientation and position.
 13. The imageforming apparatus of claim 12, wherein the mechanism includes at leastone of a gonio system and a moving stage.
 14. The image formingapparatus of claim 12, wherein the mechanism includes an optical systemwhich increases the degree of parallelism of the illuminating lightbeam.
 15. The image forming apparatus of claim 11, wherein theillumination system includes an optical system which increases thedegree of parallelism of the illuminating light beam.
 16. An imageforming method comprising: sequentially emitting illuminating lightbeams from multiple different irradiation directions with respect to anobject and irradiating the object with the illuminating light beams;capturing a plurality of different images in the multiple differentirradiation directions, respectively, using an image sensor which isarranged at a position where the illuminating light beams that have beentransmitted through the object are incident; forming a high-resolutionimage of the object, having a higher resolution than any of theplurality of images, based on the plurality of images; and storing in amemory data about the ratio of light rays that have been incident on aphotoelectric conversion section of each pixel of the image sensor tolight rays that have passed through the upper surface of a plurality ofsubpixels included in the pixel with respect to each of the multipleirradiation directions, wherein the forming a high-resolution image ofthe object includes the forming the high-resolution image of the objectby multiplying the inverse matrix of a matrix having the ratio as itselement by the vector of the pixel values based on the data that hasbeen retrieved from the memory by extracting, as a vector, a set ofpixel values associated with the multiple irradiation directions frompixel values that form each of the plurality of images.
 17. An imagesensor for use in an image forming apparatus, the image formingapparatus comprising: i) an illumination system which sequentially emitsilluminating light beams from multiple different irradiation directionswith respect to an object and irradiates the object with theilluminating light beams; ii) a holder which holds the image sensor inan attachable and removable state; iii) an image processing sectionwhich forms, based on a plurality of different images that have beencaptured by the image sensor in the multiple different irradiationdirections, respectively, a high-resolution image of the object having ahigher resolution than any of the plurality of images; and iv) a memorywhich stores data about the ratio of light rays that have been incidenton a photoelectric conversion section of each pixel of the image sensorto light rays that have passed through the upper surface of a pluralityof subpixels included in the pixel with respect to each of the multipleirradiation directions, the image processing section forming thehigh-resolution image of the object based on the data that has beenretrieved from the memory by extracting, as a vector, a set of pixelvalues associated with the multiple irradiation directions from pixelvalues that form each of the plurality of images, the image sensor beingarranged attachably to, and removably from, the image forming apparatus,an imaging surface of the image sensor has an object supporting portionwhich is a region on which the object is able to be mounted, and theimage sensor is arranged at a position where the illuminating lightbeams transmitted through the object are incident while being held bythe holder onto the image forming apparatus and captures the pluralityof different images in the multiple different irradiation directions.18. The image sensor of claim 17, wherein the image sensor is arrangedabove slide glass, and held by the holder onto the image formingapparatus attachably to, and removably from, the image forming apparatusvia a portion of the slide glass.
 19. The image sensor of claim 17,wherein an opaque member is arranged on a side surface of the imagesensor.
 20. The image sensor of claim 17, wherein an opaque region whichlimits an image capturing range is arranged on the object supportingportion.